nvee_backend.md

Go to the documentation of this file.

Copyright (c) 2016, NVIDIA CORPORATION.  All rights reserved.

@page nvvid_backend_group 4 Ways to Video Decode with GIE Processing Sample
@{

 - [Overview](#overview)
 - [Key Structure and Classes](#key)
 - [Command Line Options](#options)

- - - - - - - - - - - - - - -
<a name="overview">
## Overview ##

This use case application showcases a typical appliance performing intelligent video analytics.
Application areas include public safety, smart cities, and autonomous machines. This example demonstrates
four (4) concurrent video streams going through a decoding process using the on chip decoders, video scaling using on
chip scalar, and GPU compute. For simplicity of demonstration, only one of the channels will use GIE to perform object identification
and generate bounding box around the identified object. This sample also uses video converter functions
to do various format convrsions and uses EGLImage to demonstrate buffer sharing and image display capability.

In this sample, object detection is limited to identifying cars in video streams of 960 x 540 resolution,
running up to 14 FPS. The network is based on GoogleNet. The inference is performed on a frame-by-frame basis
and no object tracking is involved. Note that this network is intended to be purely an example to showcase
GIE usage to build the compute pipeline quickly. The trained GoogleNet is provided as source and was trained
using NVIDIA DIGITS with roughly 3000 frames taken from 5-10 feet elevation. Varying levels of detection accuracy are expected based on the video samples fed in. Given that this sample is locked to perform at half-HD resolutions under 10 FPS, video feeds with higher FPS for inference will show stuttering during playback.

`NvEGLImageFromFd` is NV-defined API to return an EGLImage pointer from the file descriptor buffer that is allocated by way of Tegra mechanism. EGLImage buffer is then used by GIE to render the bounding box to the image.

This sample does not require a camera.

The image below shows a sample block diagram.

![ ](nvee_backend_one.jpg)

The images below shows data flow details for the channel using GIE.

![ ](nvee_backend_two.jpg)

`NvEGLImageFromFd` is NVIDIA defined API to return an `EGLImage` pointer form the file
descriptor buffer that is allocated by way of Tegra mechanism. `EGLImage` buffer is
then used by GIE to render the bounding box to the image.

#### Prerequisites ####
Before running the sample, you must have the following:

* CUDA 8.0 Toolkit for L4T Rel 24.2
* GPU Inference Engine (GIE)
* OpenCV4Tegra
* README that provides details on the environment requirements to build and run the sample

- - - - - - - - - - - - - - -
<a name="key">
## Key Structure and Classes ##

There is a global structure `struct context_t` that manages all the resources in the application.

|Element|Description|
|-------|-----------|
|[NvVideoDecoder](classNvVideoDecoder.html)|Contains all video decoding-related elements and functions.|
|[NvVideoConverter](classNvVideoConverter.html)|Contains elements and functions for video format conversion.|
|NvEglRenderer|Contains all EGL display rendering-related functions.|
|egl_image|The EGLImage used for CUDA processing.|
|conv_output_plane_buf_queue|Output plane queue for video conversion.|
|dec_capture_loop|The thread handler for decoding capture loop.|

### %NvVideoDecoder ###

The [NvVideoDecoder](classNvVideoDecoder.html) class creates a new V4L2 Video Decoder.
The following table describes the key %NvVideoDecoder members that this sample uses.

|Member|Description|
|-------------|---|
|[output_plane](classNvV4l2Element.html#aaba251827cef1b23e7c42f776e95fee5) |Holds the V4L2 output plane.|
|[capture_plane](classNvV4l2Element.html#a91806d7ed13b4b2c48758e8a02f46c6d)|Holds the V4L2 capture plane.|
|createVideoDecoder  |Static function to create video decode object.|
|subscribeEvent|Subscribes event.|
|setExtControls |Sets external control to V4L2 device.|
|setOutputPlaneFormat|Sets output plane format.|
|setCapturePlaneFormat|Sets capture plane format.|
|getControl|TBD|
|dqEvent|Dequeues the devent reported by the V4L2 device.|
|isInError|Checks if under error state.|

### %NvVideoConverter ###

The [NvVideoConverter](classNvVideoConverter.html) class packages all video converting related elements and functions.
It performs color space conversion, scaling and
conversion between hardware buffer memory and software buffer memory.
The following table describes the key %NvVideoConverter members that this sample uses.

|Member|Description|
|-------------|---|
|[output_plane](classNvV4l2Element.html#aaba251827cef1b23e7c42f776e95fee5) |Holds the output plane.|
|[capture_plane](classNvV4l2Element.html#a91806d7ed13b4b2c48758e8a02f46c6d)|Holds the capture plane.|
|[waitForIdle](classNvVideoConverter.html#a2f5d1a234427862adf9cae7323b5e24c)  |Waits until all the buffers queued on the output plane are converted and dequeued from the capture plane. This is a blocking method.|
|[setCapturePlaneFormat](classNvVideoConverter.html#ad91e70f0e8b5f5a8b4deefa5a26ffe85)|Sets the format on the converter capture plane.|
|[setOutputPlaneFormat](classNvVideoConverter.html#a38b53f4d1e2c357360893756f417faf6) |Sets the format on the converter output plane.|

Both `NvVideoDecoder` and `NvVideoConverter` contains 2 key elements: `output_plane` and `capture_plane`.
This object is dervied from class type [NvV4l2ElementPlane](group__ee__nvv4lelementplane__group.html).

### %NvV4l2ElementPlane ###

[NvV4l2ElementPlane](group__ee__nvv4lelementplane__group.html) creates an [NVv4l2Element](classNvV4l2Element.html) plane.
The following table describes the key %NvV4l2ElementPlane members used in this sample.

|Member             |Description|
|-------------------|---|
|[setupPlane](classNvV4l2ElementPlane.html#a89959f455e5222f686187cc826b1b345)         |Sets up the plane of V4l2 element.|
|[deinitPlane](classNvV4l2ElementPlane.html#af89cfe87d8f818beb0478bcf5b72574c)        |Destroys the plane of V4l2 element.|
|[setStreamStatus](classNvV4l2ElementPlane.html#a03164dde4d7ab41f3e92b41e13059316)    |Starts/Stops the stream.|
|[setDQThreadCallback](classNvV4l2ElementPlane.html#a37f213325e0e4857180f5b2319317d6a)|Sets the callback function of the `dqueue` buffer thread.|
|[startDQThread](classNvV4l2ElementPlane.html#a31f77f5e5ed1f320caa44a868a7cbedd)      |Starts the thread of the `dqueue` buffer.|
|[stopDQThread](classNvV4l2ElementPlane.html#aa798d14493de321fa90aeab6d944ca87)       |Stops the thread of the `dqueue` buffer.|
|[qBuffer](classNvV4l2ElementPlane.html#af4d52964fcfd37082f47682e457f5e95)            |Queues a V4l2 buffer from the plane.|
|[dqBuffer](classNvV4l2ElementPlane.html#a8dfcbc666ee6f36a02abfb1170ae05cd)           |Dequeues a V4l2 buffer from the plane.|
|[getNumBuffers](classNvV4l2ElementPlane.html#ac5cd394a7e0a4afd69395759aeac8787)      |Gets the number of the V4l2 buffer.|
|getNumQueuedBuffers |Gets the number of the V4l2 buffer in the queue.|
|[getNthBuffer](classNvV4l2ElementPlane.html#a868d438908f3d267dd4af1033133892f)       |Gets the \c %NvBuffer queue object at index N.|

### %GIE_Context ###

GIE_Context provides a series of interfaces to load Caffe model and perform inference.
The following table describes the key %GIE_Context members used in this sample.

|GIE_Context|Description|
|-----------|-----------|
|gie_alloc|Sets up all resources and parameters.|
|gie_free|Clears all resources.|
|caffeToGIEModel|Loads Caffe mode to GIE model.|
|doInference|Interface for inference after GIE model is loaded.|
|parse_bbox|Parses Google net's output results.|
|parse_hel_bbox|Parses Nvhel net's output results.|
|parse_net|Parses net info from proto file.|

2 global functions are used to create and destroy EGLImage from `dmabuf` file descriptor.

|Global Function|Description|
|---------------|-----------|
|NvEGLImageFromFd|Creates EGLImage from dmabuf fd.|
|NvDestroyEGLImage|Destroys the EGLImage.|

- - - - - - - - - - - - - - -
<a name="options">
## Command Line Options ##

To run the sample, execute:

    backend <channel-num> <in-file1> <in-file2>... <in-format> [options]

The following video formats are supported for use with command line options:

 * H.264
 * H.265

### Options ####

The table below describes the available options.

|Command|Description|
|---------------|-----------|
|-h, --help|Prints the help text.|
|-fps <fps>|Displays rate in frames per second. Default = 30.|
|--s| Provides a statistic of each channel.|
|-run-opt <0-3>| 0=default, 1=parser only, 2=parser + decoder, 3=parser + decoder + VIC|
|--input-nalu|Specifies that the default input to the decoder is in nalu units.|
|--input-chunks|Specifies that the default input to the decoder is in chunk of bytes.|
|--gie-deployfile|Sets the deploy file name.|
|--gie-modelfile|Sets the model file name.|
|--gie-proc-interval|Sets the process interval; 1 frame is processed every gie-proc-interval.|
|--gie-float16|0=default, 1=float16, 2=float32.|

For X11 technical details, see:

http://www.x.org/docs/X11/xlib.pdf